Improving knowledge distillation using unified ensembles of specialized teachers

The increasing complexity of deep learning models led to the development Knowledge Distillation (KD) approaches that enable us transfer knowledge between a very large network, called teacher and smaller faster one, student. However, as recent evidence suggests, using powerful teachers often negatively impacts effectiveness distillation process. In this paper, reasons behind apparent limitation are studied an approach transfers more efficiently is proposed. To end, multiple highly specialized employed, each one for small set skills, overcoming aforementioned limitation, while also achieving high efficiency by diversifying ensemble. At same time, employed ensemble formulated in unified structure, making it possible simultaneously train models. proposed method demonstrated three different image datasets, leading improved performance, even when compared with state-of-the-art ensemble-based methods.